I 


PREDICTING 


OPERATOR 


N88 - 2337 


WORKLOAD DURING SYSTEM DESIGN 


$3 

790793 

;i(r 


Theodore B. Aldrich and Sandra M. 
Anacapa Sciences, Inc. 
Fort Rucker, Alabama 


INTRODUCTION 



Current Army policy requires that human capabilities and 
limitations be addressed during the conceptual phase of new 
weapon systems development. In furtherance of this policy, 
Anacapa Sciences, Inc. researchers, under contract to the 
U.S. Army Research Institute Aviation Research and Develop- 
ment Activity (ARIARDA) , developed a methodology to predict 
aviator workload in advance of aircraft system design. The 
methodology features models that predict workload under vary- 
ing automation configurations for both single- and multi-crew 
system designs. This paper (a) describes the methodology for 
developing and exercising the workload prediction models and 
(k) presents flight simulator— based research plans for 
validating the workload predictions yielded by the models. 


THE WORKLOAD PREDICTION METHODOLOGY 
Background 

The Army's Air/Land Battle 2000 scenario represents a 
high— threat environment that will place heavy workload 
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systems are being developed with advanced technology designed 
to automate many of the functions traditionally performed by 
crew members. Examples of the advanced technology include: 

* an increased number of sensors and target acquisition 
aids 


• improved navigation and communication systems 

• advanced crew station design features 

• improved flight controls 

• extraordinary avionics reliability 

• subsystems that are automatically reconfigured if 
components fail 

Although advanced technology is typically designed to 
reduce aviator workload, the tasks required to use the 
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technology may actually increase workload in some instances. 
For example, technology designed to reduce an aviator s nee 
to maintain physical control of system functions often 
increases the aviator's role as a systems monitor or problem 
solver. Consequently, while psychomotor workload demands are 
decreased, sensory and cognitive attentional demands are 
increased . 


The development of new and improved aircraft systems 
also presents problems in the prediction and assessment of 
operator workload. Metrics that are appropriate for analyz 
ing physical workload are inadequate for assessing sensory 
and cognitive workload. Accordingly, workload research has 
shifted from a focus on physical effort required to perform a 
task to an emphasis on the attentional demand associated with 
the sensory, cognitive, and psychomotor workload components 
of the tasks. The workload prediction methodology developed 
by ARIARDA and Anacapa researchers operationally defines 
workload in terms of attentional demand. Consequently, the 
methodology is designed to measure "mental state" associated 
with task performance. 

The workload prediction methodology was developed in 
response to a request for research support from the Army s 
Aviation Systems Command (AVSCOM) Program Office charged with 
the development of a new multipurpose, lightweight hell 
copter, designated the LHX . A detailed description of the 
manner in which the methodology was developed and applied to 
the LHX is presented in reference 1. 


The original LHX workload prediction methodology cur- 
rently is being refined during analyses of three additional 
Army helicopter systems and one advanced-technology crew 
station for an experimental research flight simulator. T e 
four additional analyses are: 

• a baseline analysis for the AH-64A, Apache, prior to 
predicting crew workload in a proposed AH-64B configu 
ration (ref. 2) 

. a baseline analysis for the UH-60A, Blackhawk, prior 
to predicting crew workload in a redesigned MH 6 X 
configuration (ref. 3) 

• a baseline analysis for the CH-47, Chinook, prior to 
predicting crew workload in a redesigned MH 47E 
conf igurat ion 

• a baseline analysis for an advanced technology LHX- 
type crew station for the Crew Station Research and 
Development Office (CSRDO) at NASA Ames, prior to 
predicting crew workload in high-fidelity flight 
simulation experiments 
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In applying the methodology to the aircraft and flight 
simulator systems, three major phases of research must be 
performed: 

conduct mission/task analyses of critical mission 
segments and assign estimates of workload for the 
sensory, cognitive, and psychomotor workload 
components of each task identified 

• develop computer-based workload prediction models 
using the data produced by the task analyses 

• exercise the computer models to produce predictions of 
crew workload under varying automation and/or crew 
conf igurat ions 

Each of the three phases in the refined methodology is 
described below: 


Phase 1: Conduct Mission/Task Analysis 

T he first phase of the methodology is to conduct a 
comprehensive mission and task analysis for the proposed 
aircraft or simulator system. The mission/task analysis uses 
a top— down approach m which mission profiles for the system 
are subdivided into mission phases, and subsequently into 
mission segments. A segment is defined as a major sequence of 
events that has a definite start and end point . The events 
in a segment may occur concurrently or sequentially. 

Each segment is then divided into functions. A function 
is defined as a set of activities that must be performed 
either by an operator or by equipment to complete a portion 
of the mission segment. Functions are categorized as contin- 
uous, discrete fixed, or discrete random and are placed on a 
rough time line using a Segment Summary Worksheet, such as the 
example selected from the AH-64A mission/task analysis 
(ref. 2) and depicted in Figure 1. 

The functions for each segment are subsequently divided 
into tasks. Each task is a specific crew activity that is 
essential to the successful performance of the function. The 
task consists of a verb and an object and is analyzed to 

• identify the crewmember (s) performing the task 

• identify the subsystem representing the primary man- 
machine interface 

• estimate the workload imposed on the crew member (s) 


• estimate the time required to complete the task 
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The crew member (s) performing each task and the 
subsystems associated with each task are identified by 
examining the manner in which similar tasks are performed in 
existing Army helicopters- Predictions of the visual, audi- 
tory, kinesthetic, cognitive, and psychomotor workload for 
each task are derived by writing short verbal descriptors of 
the requirements for each task component . The descriptors 
are then compared with the verbal anchors contained in the 
rating scales shown in the table (ref. 2) . The rating (i.e., 
1-7) associated with the anchor that best matches the 
verbal descriptor is assigned as the numerical estimate of 
workload. Two or more analysts perform the ratings inde- 
pendently and then reach consensus on the final ratings for 
each task. Task time estimates are assigned after interviews 
with subject matter experts (SMEs) , or in some cases, after 
actual measurements of performance times on similar tasks. 

Information derived from the mission/task and workload 
analyses is recorded on Function Analysis Worksheets, such as 
the one shown in Figure 2 for the AH-64A function "Fire 
Weapon, Missile" (ref. 2) . The tasks are listed in the first 
two columns. The crew member performing each task is indi- 
cated by the letter (P for pilot; G for gunner; and B for 
both) that is presented in the third column along with a 
numerical identifier for the task. The subsystems associated 
with each task are presented in the fourth column. Verbal 
descriptors of the sensory, cognitive, and psychomotor com- 
ponents of workload and the ratings associated with each 
component are entered in the next three columns . The eighth 
column describes the type of switch for each task for which a 
specific switch is involved - . The estimated length of time 
for discrete and continuous tasks is presented in the final 
two columns of the worksheet . The total time to perform all 
the tasks in the function appears in the upper right corner 
of the Function Analysis Worksheet . 


Phase 2: Develop Computer-Based Workload Prediction Models 

Phase 2 of the methodology consists of developing 
computer models to predict total workload experienced in the 
performance of both individual and concurrent tasks. The 
procedure used to develop the computer models represents a 
bottom— up approach in which the tasks identified in the Phase 
1 mission/task analysis serve as the basic elements of 
analysis. Specifically, the information derived for each 
task is entered into computer data files from which estimates 
of total workload at the segment level are produced. 

Computer programs developed from time-based decision rules 
are then written to build functions from the tasks, and 
subsequently, to build segments from the functions. The 
decision rules define the temporal relationships among tasks 
and functions as determined in the mission/task analysis. By 
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implementing the decision rules, the computer models produce 
estimates of total workload, at half-second intervals, for 
each workload component (i.e., visual, auditory, kinesthetic, 
cognitive, and psychomotor) . The estimates are derived by 
summing the ratings assigned to each workload component 
across concurrent tasks. A total value of "8" on any single 
half-second time line constitutes the threshold for an 
overload within a given workload component. A more detailed 
description of the Phase 2 methodology is provided in 
references 1, 2, and 3. 


Phase 3: Exercise the Computer Models 

During Phase 3, the computer models are exercised to 
predict workload associated with individual automation 
options and/or combinations of options. Three steps are 
performed to produce the workload predictions: 

• select the automation options to be exercised by the 
model 

• revise the estimates of workload for each task 

• exercise the model to produce new workload 
predictions 

The automation options are selected in consultation with 
engineers from the system program office responsible for 
acquiring the new aircraft or flight simulator. The tasks 
identified in the mission/task analysis are then reviewed to 
determine how each of the proposed automation options is 
likely to change the workload estimates in the baseline 
analysis. For each task affected by the automation options, 
new verbal descriptors of workload are written. These 
descriptors, in turn, provide the basis for assigning new 
workload ratings to the components of the tasks . New computer 
files containing the revised workload estimates are then 
established. Finally, the model is exercised with the new 
files to predict workload for any single automation option or 
any combination of automation options. Use of the model to 
predict crew workload for the LHX weapon system is described 
in detail in reference 1. 


Application of the Workload Prediction Methodology 

The methodology described above represents a systematic 
approach for predicting operator workload in advance of 
system design. As various automation options and alternative 
crew configurations are considered during the design of a 
weapon system, the methodology can be repeated so that the 
workload predictions keep pace with the system design 
process. Additionally, the methodology produces a number of 



products that can be applied to the development of any 
complex weapon system. The products include: 

• a mission/task/workload analysis that provides 
estimates of (a) sensory, cognitive, and psychomotor 
components of workload, and (b) performance times at 
the task level of specificity 

• scales for rating sensory, cognitive, and psychomotor 
components of workload 

• a timeline analysis that depicts concurrent crew 
tasks 

• a procedure for evaluating total workload for 
concurrent crew tasks 

• a numerical index for identifying crew overloads 

• computer models that produce comparisons of workload 
for proposed alternatives in system design and crew 
composition 

• a procedure for identifying an optimum design 
configuration for reducing crew workload 

Workload predictions produced by the models have already 
been used by the Army in system trade-off analyses directed 
toward determining whether one or two aviators will be 
required to perform the LHX mission on the future battlefield 
and to assist in making decisions regarding the optimum 
configuration of LHX automation options. 


VALIDATION OF THE WORKLOAD PREDICTION MODEL 

The workload predictions yielded by the models have not 
been validated. Consequently, the next phase of the research 
will consist of (a) validation of the parameters used to 
develop the models, and (b) the validation of the workload 
predictions yielded by the models. 

Parameters of the model that require validation include: 


• workload ratings assigned to each task 

• total workload estimates for concurrent tasks 

• estimated times assigned to each task 

• threshold for excessive workload 

• temporal relationships among tasks 

• procedural relationships among tasks 


In designing the validation research a number of 
critical issues were considered. In this section, two of 
the critical issues most relevant to the workshop topic, 
Mental— State Estimation , are discussed and major provisions 
of the validation research plan are presented. A more com- 
plete discussion of the critical issues and a full descrip- 
tion of current research plans are presented in reference 4 


Critical Issues 

The problems and issues that have a critical bearing on 
the research required to validate the parameters in the 
workload prediction methodology include the following: 

• reliability and validity of workload predictors 

• selection of appropriate criterion measures. 

Reliability and Validity of the Workload Predictors 

The methodology used to derive the workload predictions 
requires that the reliability of both the rating scales and 
the predictors of workload be established. Specifically, it 
must be demonstrated (a) that the workload rating scales 
discriminate accurately between levels of attentional demand, 
and (b) that different raters will derive consistent esti- 
mates of workload for the sensory, cognitive, and psychomotor 
components of individual tasks. The reliability of the 
ratings assigned to the individual task components is 
important because these ratings are the basis for producing 
the predictors of total workload for concurrent tasks. If the 
individual workload ratings are found to have high reliabil- 
ity, the predictors of total workload produced by summming 
the ratings also will have high reliability. 

The procedures used to develop the workload predictors 
are designed to ensure that the predictors have high face and 
content validity. The research for validating the workload 
model will attempt to establish that the predictors also have 
predictive validity. The predictive validity will be 
established by comparing the workload component ratings for 
each task, as well as the predictions of total workload 
associated with concurrent tasks, with (a) objective measures 
of primary task performance and (b) other subjective measures 
of workload. The primary task measures will be compared with 
the predictors at half-second intervals for each task on the 
mission segment timeline, while the subjective measures will 
be compared with the predictors for selected portions of the 
mission segments. Predictive validity will be demonstrated 
to the extent that the workload component ratings and/or the 


total workload predictors correlate with the criterion 
measures . 


Selection of Appropriate Criterion Measures 

A number of performance measures will be selected as 
criteria for validating the workload predictors. Although 
evidence suggests that, in some instances, task performance 
may be relatively independent of workload (ref. 5), a criti- 
cal assumption of the workload prediction model is that, when 
total attentional demand is driven close to or above the 
threshold of overload, performance on one or more of the 
concurrent tasks will be degraded. Consequently, the primary 
basis for selecting the performance measures to be used in 
the validation study will be their sensitivity to degrada- 
tions in task performance due to increased workload. Addi- 
tionally, the measures will be selected on the basis of their 
relevance to specific operator tasks. For example, devia- 
tions from a specified airspeed will be the criterion for 
workload encountered in the task "control airspeed." Such 
measures have high face, content, and construct validity. 

Subjective measures of workload also will be collected 
during the validation research. The subjective measurements 
will be selected from among presently recognized and 
partially validated techniques, including (a) the NASA 
bipolar rating technique (ref. 6), (b) a modified Cooper- 

Harper rating technique (ref. 7), and (c) the subjective 
workload asessment technique (SWAT) (ref. 8). 

Subjective measurements offer the system designer 
information that is not provided by the more objective 
techniques; furthermore, subjective methods of measurement 
are generally well received by operators and require little 
instrumentation. The greatest disadvantage of subjective 
workload measurements from the standpoint of the validation 
research is that the measurements do not provide information 
regarding the composition of the primary task. That is, it is 
just not feasible to collect subjective ratings at the task 
level of specificity. A second disadvantage is that 
subjective methods rely on the ability of operators to 
retrieve information from short-term and long-term memory 
regarding their experiences during task execution; yet, the 
behavioral literature is replete with examples demonstrating 
the fallibility of the memory retrieval processes (refs. 9 
and 10) . Even if the retrieval processes were reliable, it 
is not clear whether the recollections reflect task input 
modality (ref. 11), number of concurrent tasks (ref. 12), 
working memory load (ref. 13), or some other aspect of the 
task situation. Finally, empirical findings (ref. 14) 
suggest that retrospective subjective measures reflect the 
average workload experienced during task execution, thus 
precluding the analysis of workload at different points in 
time . 


For several reasons there presently are no plans to 
employ physiological workload measurement techniques during 
the validation research. No single physiological measurement 
technique exists that is sensitive to task loading, 
diagnostic of task demand, and unobtrusive. A more serious 
problem with physiological measures is that they do not 
directly address the relationship between system design and 
workload, an important consideration on which system 
engineers base their design decisions. There are simply not 
enough data to establish whether the fluctuations of 
physiological measures actually reflect mental effort, some 
other operator "state" condition such as stress or fatigue, 
or a combination of several workload-related states. 


The Validation Research Plan 

The proposed research for validating the workload 
prediction methodology will be accomplished in three phases. 
During Phase 1, the reliability of the workload rating scales 
and the workload predictors will be evaluated. During Phase 
2, validation data will be collected through a series of 
studies employing part-mission and full-mission simulation. 
During Phase 3, the results from Phases 1 and 2 will be used 
to refine the workload prediction model. Each of the three 
phases are described briefly below. More complete details 
are provided in reference 4. 

Phase 1; Establish the Reliability of the Workload 

Rating Scales and the Workload Predictors 

Phase 1 of the validation research will evaluate how 
closely the researchers' judgments in assigning numbers to 
the verbal anchors correspond with the judgments of other 
human factors scientists engaged in workload research. 

First, a psychophysical experiment using the method of paired 
comparisons (ref. 15) will be conducted by survey to (a) 
verify the ordinal ranks of the verbal anchors for each of 
the five workload component scales, and (b) produce equal 
interval scale values for each verbal anchor. Second, the 
empirically derived interval scale values will be applied to 
the workload component descriptors for all tasks. Finally, 
predictors of total workload will be produced by summing the 
interval scale values across concurrent tasks. 

The human factors scientists also will be requested to 
rate the short descriptors of visual, auditory, kinesthetic, 
cognitive, and psychomotor components of workload for each 
task in the model. These same judges subsequently will be 
teamed in pairs. Each pair of judges will be instructed to 
assign a consensus rating for each of the verbal descriptors. 
Correlational techniques will be used to evaluate the 



inter-rater reliability of the ratings produced by (a) each 
independent rater and (b) each pair of raters. 

Phase 2: Conduct Part-Mission and Full-Mission Simulation 

During Phase 2 of the validation research, both part- 
mission and full-mission simulation experiments are planned. 
The simulator configuration for both the part-mission and the 
full-mission simulation will be identical. For the part- 
mission simulation, mini-scenarios will be generated by 
selecting concurrent and sequential tasks from the 
mission/task analysis. An equal number of the mini-scenarios 
containing high- and low-workload sets of tasks will be 
selected. For the full-mission simulation, a composite 
mission scenario will be developed by selecting segments from 
the mission/task analysis. 

The part-mission simulation will be conducted using a 
repeated measures experimental design in which each subject 
will fly the mini-scenarios multiple times. The order of 
presentation of the mini-scenarios will be counterbalanced to 
control for order effects and other extraneous variables. 
Analyses will then be performed to assess the correlation 
between the workload predictors and the performance measures 
recorded throughout the mini-scenarios. The correlation 
coefficients resulting from the analyses will serve as the 
primary measure of how accurately the workload predictors 
forecast excessive workload at the task level of specificity. 
Analyses also will be performed to assess the correlation 
between predictions of workload and subjective estimates of 
workload. These correlations will indicate the degree to 
which the workload prediction model predicts workload at the 
mini-scenario level of specificity. 

To assess the validity of the time estimates used in the 
model, the actual amount of time required to perform the 
various tasks in the mini-scenarios will be compared with the 
estimated times produced during the task analysis . 

Differences will be resolved by adopting the recorded times. 
The time analysis will be used to validate the temporal 
relationships among the tasks as they exist in the workload 
prediction model. The procedural relationships among the 
tasks will be evaluated by noting the subjects' ability to 
progress through the mini-scenarios following the sequence of 
tasks specified by the model. Any new sequences adopted by 
the subjects to complete the mini-scenarios will be used to 
refine the workload prediction model. 

During the full-mission simulation experiments, each 
trial will start at the beginning of the composite scenario 
and continue without interruption to the end. The analysis 
of results from the full-mission simulation will include all 
of the analyses performed during the part-mission simulation 
data analysis. 
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Phase 3: Refine the Workload P rediction Model 


The final phase of the validation research will be to 
refine the workload prediction model. The first refinements 
will be made when the research results from Phase 1 are 
available. Additional refinements will be made when the 
part-mission simulation results are available; final 
refinements will be made when the full-mission simulation 
results are available. 


CONCLUSIONS 

Successful completion of the validation research will 
result in several useful products. The products will include 
(a) reliable and valid scales for predicting visual, audi- 
tory, kinesthetic, cognitive, and psychomotor workload at the 
task level of specificity, and (b) a validated workload pre- 
diction methodology that can be applied early in the system 
design process. Even without validation, the workload pre- 
diction methodogy proved useful during the trade-off analyses 
and other system studies conducted for the LHX. The baseline 
analyses currently being performed for the AH-64A, UH-60A, 
and CH-47 aircraft will benefit proposed modification pro- 
grams for additional systems. After the validation research 
has been completed, the human factors community will have a 
tool with proven value for predicting operator workload early 
in the design of any proposed system. 
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WORKLOAD COMPONENT SCALES 


SCALE 

VALUE DESCRIPTORS 


Cognitive 

1 Automatic (Simple Association) 

2 Sign/Signal Recognition 

3 Alternative Selection 

4 Encoding/Decoding, Recall 

5 Evaluation/Judgment (Consider Single Aspect) 

6 Evaluation/Judgment (Consider Several Aspects) 

7 Estimation, Calculation, Conversion 

Visual 

1 Visually Register/Detect (Detect Occurrence of Image) 

2 Visually Inspect/Check (Discrete Inspection/Static Condition) 

3 Visually Scan/Search/Monitor (Continuous/Serial Inspection, Multiple Conditions) 

4 Visually Locate/Align (selective Orientation) 

5 Visually Track/Follow (Maintain Orientation) 

6 Visually Discriminate (Detect Visual Differences) 

7 Visually Read (Symbol) 

Auditory 

1 Orient to Sound (General Orientation/Attention) 

2 Orient to Sound (Selective Orientation/Attention) 

3 Detect/Register Sound (Detect Occurrence of Sound) 

4 Verify Auditory Feedback (Detect Occurrence of Anticipated Sound) 

5 Discriminate Sound Characteristics (Detect Auditory Differences) 

6 Interpret Semantic Content (Speech) 

7 Interpret Sound Patterns (Pulse Rates, etc.) 

Kinesthetic 

1 Detect Preset Position/Status 

2 Detect Movement (Discrete Actuation-Toggle, Trigger, Button) 

3 Detect Movement (Discrete Adjustive-Rotary Switch) 

4 Detect Movement (Continuous Adjustive/Flight Controls-Cyclic, Collective) 

5 Detect Movement (Continuous Adjustive/Switches-Rotary Rheostat, Thumbwheel) 

6 Detect Serial Movement (Keyboard Entries) 

7 Detect Conflicting Cues 

Psychomotor 

1 Discrete Actuation (Button, Toggle, Trigger) 

2 Discrete Adjustive (Rotary, Vertical Thumbwheel, Lever Position 

3 Speech 

4 Continuous Adjustive (Flight Control, Sensor Control) 

5 Manipulative 

6 Symbolic Production (Writing) 

7 Serial Discrete Manipulation (Keyboard Entries) 
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SEGMENT SUMMARY WORKSHEET 


PHASE 3 Enroute SEGMENT 08 Takeoff 


PILOT 

GUNNER 

DISCRETE (FIXED) 

DISCRETE 

(RANDOM) 

CONTINUOUS 

DISCRETE (FIXED) 

DISCRETE 

(RANDOM) 

CONTINUOUS 

Perform Hover 
(100) 

Perform Before 
Takeoff Check 
(091) 

Perform 

External 

Communication 

(099) 

Establish Climb 
(059) 

Establish Level 
of Flight (060) 

Receive 
Communication 
(Internal) (116) 

Transmit 
Communication 
(Internal) (148) 

Monitor Audio 
(078) 

Perform Before 
Takeoff Check 
(090) 

Receive 
Communication 
(Internal) (116) 

Transmit 
Communication 
(Internal) (148) 

Monitor Audio 
(078) 


Figure 1 . Example of a Segment Summary Worksheet developed during the mission/task analysis (ref. 2). 



FUNCTION ANALYSIS WORKSHEET 


FUNCTION 065 Fire Weapon, Missile TOTAL TIME (Approximate) 5.5 Seconds 


TASKS 



WORKLOAD COMPONENTS 


DURATION 

(SECONDS) 

VERB 

OBJECT 

ID # 

SUBSYSTEM(S) 

SENSORY 

COGNITIVE 

PSYCHOMOTOR 

SWITCH 

DESCRIPTION 

DISCRETB 

CONTINUOUS 

Verify 

Firing Constraints 

G239 

Sensor Display 
(VSD) 

Visually Discriminate 
Alignment Differences 
V-6 

Evaluate Sensory 
Feedback and Verify 
Constraints Met 
C-2 



1 


Pull 

Weapons Trigger 

B643 

Weapons 

(AW) 

Feel Trigger Movement 
K-2 

Verify Correct Position 
(Trigger Activated) 

C-2 

Lift Cover and Pull 

Trigger 

P-1 

Spring loaded 

Trigger 

(SPTR) 

1 


Verify 

Missile Launch 

G417 

Fire Control 
Computer/ 
Sensor Display 
(AFC/VSD) 

Visually Detect Image 
V-1 

Verify Correct Status 
(Missile Launched) 
C-2 



1 


Release 

Weapons Trigger 

B644 

Weapons 

(AW) 

Feel Trigger Movement 
K-2 

Verify Correct Position 
(Trigger Deactivated) 
C-2 

Release Trigger 
P-1 

Springloaded 

Trigger 

(SPTR) 

.5 



Figure 2. Example of a Function Analysis Worksheet developed during the mission/task analysis (rel. 2). 
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